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ABSTRACT 

This paper describes the development of an instrument to 
assess the kinds of information literacy learning that take place in online 
courses. It includes the following sections: (1) Overview; (2) Variables; (3) 

Blueprint; (4) Validity; (5) Reliability; (6) Interpretations; (7) 
Bibliography; and (8) See the Instrument. The author hopes that developing 
the assessment tool and evaluating the results of validity and reliability 
tests will be helpful in designing specific course content which will help 
students to build their information literacy skills for use in all kinds of 
learning. The assessment might also be helpful in teacher training as 
teachers become interested in using computer technologies in their classes, 
both on campus and online. Also, development of appropriate assessment tools 
will help make educators and the public aware of a special value of online 
education: to develop necessary information-handling skills that students 
will need for future employment and academic work. The paper gives a brief • 
overview of the instrument development project, discusses the variables used 
in assessing growth in information literacy, provides a blueprint for the 
information literacy achievement continuum, details the results of the 
validity and reliability tests, interprets the factors which influenced the 
results of the pilot study, and provides recommendations for modifications to 
the informatioh literacy instrument. (CB) 
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Teachers who use in-class group work, for example, add communication skills to the list of things students learn in 
class. When students are truly engaged in that process, they may learn more about communication than others do. 
Another example is teachers who assign reseiirch projects. Students will invariably learn something about research 
methods and techniques, but students who are more deeply interested in the subject matter may be more stimulated 
than others to learn how to conduct more complex research activities. 
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The five variables I ultimately plan to assess are: tool literacy, resource/social-structural literacy, research literacy, 
publishing literacy, and emerging technology literacy. For this one-semester project, I focus on the Tool Literacy 
variable. To put it in context, the following five tables give brief descriptions of all the variables and suggest general 
indicators of achievement for each (see Tables 1-5). 
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Table 2: Resource/Social-Structural Literacy Variable 
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Table 5: Emerging Technology Literacy Variable 
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the conference, but may overlook an appropriate pre-existing conference topic. Students should be able to 
describe why they used the search engines. Low level responses will use a CD reference only, or only one 
search engine. Students may not be able to post a message to the conference area, and don't have a good 
idea of why they used the tools because they are imfamiliar with the options available to them. 
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Activity 


Submit the survey. 


Send an email to a specified address. 


Read an email response, open an attachment, and forward the email to 
another specified address. 


Participate in an existing online conference by posting a message to the 
pre-existing topic "Survey Task 2." 


Participate in an online conference by creating a new topic area with a 
question. 


Respond to postings in the new topic area by searching for information 
on a specified topic and describing your search. 


Differentiate between using email and using an online conference. 


Subscribe to a listserver and describe how you subscribed to it. 


Find a listsever about a specified topic and describe how you found it. 


Differentiate between using a conference and a listsever. 


Differentiate between using email and a listsever. 
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Table 6: Items 
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The instrument was designed to be used as diagnostic tool early in the course. Performance on the 
instrument did support my hypothesis about difficulty (see "Content Validity" below), and the item 
breakdown was sufficient to identify specific areas where students could improve, so it looks like a good 
diagnostic tool. 



Content was derived from discussions with faculty and review of proposed standards for information 
literacy. 
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This at least places items 5 and 7 at an equivalent level of difficulty. It moves item 3 
(forwarding email), however, to a much higher ranking than expected. Ultimately, these 
results are suspect because of the skewing toward high-performing respondents, problems 
encountered submitting the forms, and the voluntary nature of responding (see Table 10). 



Participants may perform well because the tasks "give away" some of the answers. In addition, these 
participants were highly motivated to demonstrate how capable they are, since they are my former 
students and are also interested in my referrals for jobs. 

[ Table Omitted ] 
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A final problem was the instrument itself, in that it did not cover advanced aspects of using online tools. 
This was highlighted as a deficiency with this particular sample because so many respondents were 
experienced users. In a "normal" group of students, I may have foimd the items appropriate. 



These problems will be resolved by using the instrument as a required pre-test instrument and redesign of 
the instrument format. Using the instrument as a pre-test will also allow me to correlate initial ability with 
student outcomes in courses. If the instrument is also used as a post-test, I can compare results to 
determine change. 
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